Stereoscopic image display method and stereoscopic image display apparatus

ABSTRACT

In a stereoscopic image imaging method which captures an image of an object to be displayed as a stereoscopic image, a three-dimensional image is generated in real time from multi-viewpoint images captured by a multi-camera, and is displayed on a viewer provided to the multi-camera, thus allowing a photographer to adjust an imaging condition. The photographer is informed of parameters, which implements a display state adjusted by an observer of a three-dimensional display while observing the three-dimensional image, via the viewer, and the photographer can capture appropriate multi-viewpoint images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation Application of PCT Application No.PCT/JP2009/066825, filled Sep. 28, 2009, the entire contents of whichare incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a stereoscopic imagedisplay method and stereoscopic image display apparatus for capturing animage of an object, and displaying a stereoscopic image.

BACKGROUND

As a display apparatus which can display a three-dimensional image(stereoscopic image), various systems are known. In recent years,especially, a system, which adopts a flat-panel type, and displays astereoscopic image without requiring any dedicated glasses and the like,is demanded more strongly. There has been developed a system which isprovided with a display panel (display device) and a parallax barrier(also called a ray control element) arranged in front of a displayscreen of the display panel (display device). The display panel displaysan image or a picture on the display screen having pixels fixed on aplane, like a direct-view or projection type stereoscopic moving imagedisplay device (for example, a liquid crystal display device or plasmadisplay device) and the parallax barrier controls rays coming from thedisplay panel to direct them toward an observer. This system is apractical system which can relatively easily display a stereoscopicimage.

A so-called parallax barrier controls rays so as to allow an observer toobserve different images depending on observation angles even when theobserver observes the same parallax barrier position. More specifically,when a right-and-left parallax (horizontal parallax) is given, slits ora lenticular sheet (cylindrical lens array) is used as the parallaxbarrier. When both the right-and-left parallax and an up-and-downparallax (vertical parallax) are given, a pinhole array or lens array isused as the parallax barrier. In this specification, one slit or onelens as a unit of the parallax barrier is called an exit pupil.

The system using the parallax barrier is further classified into atwo-view system, multi-view system, ultra-multi-view system(ultra-multi-view conditions are given to the multi-view system), andintegral imaging (to be simply referred to as “II” hereinafter) system.The basic principle of these systems is substantially the same as astereoscopic photo system invented about 100 years ago. However, sincethe number of pixels of a display device is finite, the number of pixelsassigned per exit pupil is also finite. In this specification, thenumber of pixels assigned per exit pupil is called the number ofparallaxes, and a two-dimensional image configured by pixels assigned torespective exit pupils is called an element image.

Note that the II system is a term of stereoscopic photography, and isalso called integral photography (to be also abbreviated as IPhereinafter).

In order to display a stereoscopic image using these II systems, images(multi-viewpoint images) captured from a plurality of directions arerequired. That is, in a stereoscopic image display method based on thetwo-view system, two multi-viewpoint images are prepared. In astereoscopic image display method based on the multi-view system or IIsystem, multi-viewpoint images as many as the number of pixelscorresponding to the number of parallaxes assigned per exit pupil areprepared. In this specification, a pixel means a minimum display unit.Basically, multi-viewpoint images are captured under the precondition ofthe relationship between pixels and exit pupils. A multi-viewpoint imagegeneration method includes a plurality of generation methods such asactual imaging and CG rendering. However, multi-viewpoint images arenormally prepared by actual imaging that captures images of an objectusing cameras.

In the actual imaging using the cameras, more specifically, cameras asmany as the number of parallaxes, which are used to capturemulti-viewpoint images, are laid out, so as to be symmetrical to arelationship between exit pupils and corresponding pixel positions. Thecameras laid out to capture multi-viewpoint images are called amulti-camera. Since pixels of a display device are arranged on a plane,the multi-camera is similarly arranged on a plane. In a stereoscopicdisplay device, letting pp be a pixel interval, and g be an intervalbetween an exit pupil and a pixel plane of the display device, animaging reference distance Lc and interval x_c of a multi-camera 1 inthe stereoscopic display device are given by:

g:pp=Lc:x _(—) c

This imaging condition means that it is most efficient to match a sizeand resolution of an imaging reference plane of the multi-camera withthose of a flat-panel display unit in the display device, so as tosatisfy the imaging condition in the multi-camera in a stereoscopicimaging device and the flat-panel display unit in the stereoscopicdisplay device. In this case, the imaging reference plane is called aprojection plane under the precondition that it is matched with thedisplay screen, the imaging reference distance is set as an observationreference visual distance of a three-dimensional display, and an imagingposition is set as a viewpoint on the observation reference plane of thethree-dimensional display. In addition, rays at the time of imaging andplayback agree with each other, and an image of an object to be capturedis displayed in a real scale.

However, this actual imaging condition need not always be strictlysatisfied. In recent years, when it is designed to observe informationof neighboring pixels to be mixed to some extent, it is devised to allowan observer to observe a stereoscopic image even outside an observationdistance range as disclosed in R. Fukushima et al., Proceedings ofSPIE-IS & T Electronic Imaging, 7237, 72370W-1 (2009). Furthermore, in athree-dimensional display based on the parallax barrier system, adisplay range in its z direction (a direction perpendicular to thedisplay screen) is limited as disclosed in J. Opt. Soc. Am. A vol. 15,p. 2059 (1998). Therefore, a multi-camera which captures multi-viewpointimages more than the number of viewpoints is prepared, multi-viewpointimages having an interval x_c smaller than a design value are selectedfrom the multi-camera, and images which are compressed in the zdirection are often displayed as disclosed in JP-A 2005-331844 (KOKAI).In this case, the z direction means a depth direction which isperpendicular to a horizontal direction x and vertical direction y of athree-dimensional display screen, and corresponds to a back surface sideof the display screen. Also, a method of displaying a stereoscopic imagewithin a display range by shifting z coordinates of existingmulti-viewpoint images upon displaying the stereoscopic image andenlarging or reducing them in the x and y directions, so as to adjustclipping ranges used as parallax images, that is, clipping methods isknown as disclosed in JP-A 2004-343290 (KOKAI). These literatures merelydisclose a display method of a stereoscopic image to be displayed byselecting already captured multi-viewpoint images or adjusting clippingranges.

In order to change z coordinates upon displaying a stereoscopic image,more specifically, a shift value for each viewpoint image within a rangeused as a parallax image need only be changed. However, in case ofactual imaging, since multi-viewpoint images are perspective projectionimages, when the projection plane is shifted forward or backward alongthe z axis upon changing the shift value, the imaging reference distanceis different from the observation reference visual distance of thethree-dimensional display, and a distortion is generated in a strictsense. In order to display a stereoscopic image free from anydistortion, the imaging reference distance to an object to be mainlydisplayed has to be set to be equal to the observation reference visualdistance of the three-dimensional display in place of the acquiredmulti-viewpoint images which have undergone post-processing, and arereconstructed to display a stereoscopic image. However, there is nomethod which allows a photographer to correctly recognize the imagingreference distance, and the imaging reference distance cannot becorrectly set. Also, there is no method which allows a photographer toknow which object an observer of the three-dimensional display locatedat a remote place wants to mainly and stereoscopically display. Sincethis object to be stereoscopically displayed does not become clear, theimaging reference distance cannot be set due to that cause.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a multi-camera imaging system used tocapture a stereoscopic image according to an embodiment, the blockdiagram also showing a layout relationship among a multi-camera, realobjects to be captured, and a projection plane.

FIG. 2 is a planar layout view showing the relationship among themulti-camera, real objects to be captured, and projection plane in thesystem for capturing a stereoscopic image shown in FIG. 1.

FIG. 3 is a planar layout view showing the relationship among a displaydevice, visible region, and observation reference plane in an apparatusfor displaying a stereoscopic image shown in FIG. 1.

FIG. 4A is a schematic plan view showing an image captured by aleft-side imaging element of the multi-camera shown in FIG. 2.

FIG. 4B is a schematic plan view showing an image captured by aright-side imaging element of the multi-camera shown in FIG. 2.

FIG. 5 is a schematic block diagram showing a stereoscopicimaging/display device according to an embodiment, which captures animage of an object, and displays it as a stereoscopic image, in themulti-camera imaging system shown in FIG. 1.

FIG. 6 is a flowchart showing a method of displaying a stereoscopicimage from images acquired by the multi-camera in the stereoscopicimaging/display device shown in FIG. 5.

FIG. 7 is a layout view for explaining a method of adjusting an imagingreference distance in the imaging method of a stereoscopic image shownin FIG. 4.

FIG. 8 is a layout view for explaining a method of adjusting an imagingreference distance in the imaging method of a stereoscopic image shownin FIG. 4.

FIG. 9 is a layout view for explaining a method of adjusting an imagingreference distance in the imaging method of a stereoscopic image shownin FIG. 4.

FIG. 10 is a schematic block diagram showing a stereoscopicimaging/display device according to another embodiment, which capturesan image of an object, and displays it as a stereoscopic image, in themulti-camera imaging system shown in FIG. 1.

FIG. 11 is a flowchart showing a method of displaying a stereoscopicimage from images acquired by the multi-camera in the stereoscopicimaging/display device shown in FIG. 10.

FIG. 12 is a schematic view showing an example of a display screendisplayed by a display unit required to display a stereoscopic imageshown in FIG. 1.

DETAILED DESCRIPTION

There will be described a stereoscopic image display method andstereoscopic image display apparatus for capturing an image of anobject, and displaying a stereoscopic image, in detail hereinafter withreference to the drawings.

According to an embodiment, there is provided a stereoscopic imagedisplay apparatus comprises a first three-dimensional display device.The first three-dimensional display device includes a first display unitconfigured to display a 2D image, the 2D image including elementalimages, and a first light control unit configured to control directionsof light rays emitted from the first display unit. The first displayunit displays each of the elemental images in a first specific areadetermined with the directions of the controlled light rays, so as todisplay a three dimensional image.

The stereoscopic image display apparatus further comprises a multicamera configured to capture multi-viewpoint images of a real objectfrom specific view points placed at certain intervals, wherein themulti-camera has a projection plane serving as an imaging referenceplane, an image processing unit configured to process images taken bythe multi camera. The image processing unit includes a parallax imagegeneration unit configured to generate parallax image data includingclip parallax images having specified ranges which are clipped from themulti-viewpoint images based on information about clipping ranges of themulti-viewpoint images, a sort processing unit configured to sort pixelsfrom the clip parallax images and rearrange the sorted pixels togenerate the elemental images, and a display condition adjustment unitconfigured to adjust parameters required to display thethree-dimensional image. The parameters are so adjusted as to capturethe real object as the multi-viewpoint images having a desired size atthe vicinity of the projection plane with reference to the displayedthree-dimensional image.

The image processing unit is configured to correct the imaging referencedistance and the intervals of the view points based on the adjustedparameters and the imaging condition to derive a corrected imagingreference distance and a corrected interval, which are required todisplay the three-dimensional image without any distortion.

In the following description of the embodiment, a parallax presentationdirection of a three-dimensional display is limited to one dimension(horizontal direction: X direction). However, the present embodiment isalso applicable to a display method and apparatus for displayingparallax information also in a direction (vertical direction: Ydirection) perpendicular to this one-dimensional direction (horizontaldirection: X direction). That is, when the present embodiment is appliedto the vertical direction (Y direction) as in the horizontal direction(X direction), parallax information can be similarly given intwo-dimensional directions (horizontal and vertical directions: X and Ydirections). Therefore, a stereoscopic image display method andapparatus according to the present embodiment not only includes anembodiment which presents parallaxes in only the one-dimensionaldirection, but also substantially includes an embodiment which presentsparallaxes also in the two-dimensional directions.

FIG. 1 shows a multi-camera imaging system, which captures astereoscopic image, according to the embodiment. FIG. 1 schematicallyshows a layout relationship between real objects 3-1, 3-2, and 3-3 to becaptured by a multi-camera 1, and a projection plane 2.

As shown in FIG. 1, with respect to the multi-camera 1 and projectionplane 2, the real object 3-1 indicated by “◯” can be laid out in frontof the projection plane 2 along a Z direction, the real object 3-2indicated by “Δ” can be laid out on the projection plane 2, or the realobject 3-3 indicated by “□” can be laid out on the back side of theprojection plane 2 along the Z direction. In this case, the multi-camera1 side with respect to the projection plane 2 is defined as the frontside, and the side opposite to the multi-camera 1 is defined as the backside of the projection plane 2. FIG. 1 shows coordinate axes (X, Y, Z)to clarify the layout relationship among the multi-camera 1, realobjects 3-1, 3-2, and 3-3, and projection plane 2. With reference tothese coordinate axes, the projection plane 2 is defined as a planewhich extends in X and Y axes perpendicular to a Z axis. In the exampleof FIG. 1, the real objects 3-1, 3-2, and 3-3 are arrayed along the Zaxis.

Data of 2D images (i.e., planar images), which are captured at a certainfield angle by the multi-camera 1 shown in FIG. 1, undergo processingrequired to display a stereoscopic image i.e., 3D image in an imageprocessing unit 40 to be converted into element image data. Data ofthese element images are supplied to a driving circuit unit 44 of aphotographer's display unit 42 provided, as a viewer device, to ahousing of the multi-camera 1. The element image data are displayed inreal time on a display panel 46, and a stereoscopic image is observed bya photographer as captured images via a parallax barrier 47. As will bedescribed in detail later, the photographer inputs an instruction froman input unit 45 to the image processing unit 40 with reference to thecaptured images. Then, an object to be captured on which thephotographer focuses interest, for example, one of the real objects 3-1,3-2, and 3-3 can be specified. An optimal imaging condition required todisplay the specified object to be captured can be set. The photographercan capture the object according to this imaging condition.

In the imaging system shown in FIG. 1, element image data from the imageprocessing unit 40 are transferred to a transmission/reception unit 58of an observation display device 52 for an observer via atransmission/reception unit 48. The imaging display device 42 isvisually confirmed by the photographer who captures an object in astudio, and is used for stereoscopic imaging. By contrast, theobservation display device 52 corresponds to a display device placed ina monitor room distant from the studio or that to be observed byobservation public who observe stereoscopic image programs. Theobserver's display device 52 includes a driving circuit unit 54 whichreceives element image data transferred to the transmission/receptionunit 58 as in the photographer's display device 42. This driving circuitunit 54 displays element images in real time on a display panel 56, andallows an observer to observe a stereoscopic image as captured imagesvia a parallax barrier 57. As will be described in detail later, theobserver inputs an instruction to the image processing unit 40 from aninput unit 55 with reference to the captured images. Thus, an object tobe captured on which the observer focuses interest, for example, one ofthe real objects 3-1, 3-2, and 3-3 can be specified. An optimal imagingcondition required to display the specified object to be captured can beset. The photographer can capture the object according to this imagingcondition.

As shown in FIG. 2, the multi-camera 1 is configured by linearlyarraying a plurality of imaging units 30-1 to 30-n as combinations oflenses 4 (4-1 to 4-n) and imaging elements 5 (5-1 to 5-n) on a planealong the X direction. Note that “n” corresponds to the number ofparallaxes, and an array interval (camera pitch) of the lenses 4-1 to4-n is called a camera interval Ls. The camera interval Ls may also begiven as an array interval (camera pitch) of the imaging elements 5-1 to5-n.

In the planar layout shown in FIG. 2, only two imaging units 30-1 and30-n corresponding to two cameras are illustrated for the sake ofsimplicity. The imaging units 30-1 and 30-n are respectively configuredby the lenses 4-1 and 4-n and imaging elements 5-1 and 5-n, and aninterval between these lenses 4-1 and 4-n and the imaging elements 5-1and 5-n is specified to be an imaging distance f. In order to capture animage (2D image) of an object at a finite distance Lc (to be referred toas an imaging distance Lc hereinafter) from the lenses 4-1 and 4-n andits surrounding space, relative positions between the lenses 4-1 and 4-nand the imaging elements 5 can be shifted to adjust imaging distances f(distances along the Z direction) between them. More specifically, thelenses 4-1 and 4-n are shifted with respect to the imaging elements 5 tochange imaging relationships of the lenses 4-1 and 4-n with respect tothe imaging elements 5, thereby changing imaging ranges. By adjustingthe imaging relationships of the lenses 4-1 and 4-n, the imaging rangesof the multi-camera 1 can overlap each other at an imaging referencedistance Lc, as indicated by hatching. The lenses 4-1 and 4-n havecertain field angles, and the imaging ranges of the multi-camera 1 arespecified by these field angles and imaging reference distance Lc.

A range of an X-Y plane at the imaging reference distance L in a spacewhere the imaging ranges overlap each other is defined as the projectionplane 2 under a given condition to be described later. Before and afterthe projection plane 2, an imaging range 6 extends, as shown in FIG. 2.Within this imaging range 6, the real object 3-1 is located on the frontside (in a projecting direction) of the projection plane 2 by a distancez_n. Or the real object 3-2 is located on the projection plane 2, or thereal object 3-3 is located on the back side (in a depth direction) ofthe projection plane 2 by a distance z_f. Then, when images of theobject are captured in this state, they can be displayed as astereoscopic image on the display device. These distances z_n and z_fwhich allow images to be displayed as a stereoscopic image correspond toa projecting display region and depth display region on the displaydevice side. That is, the distances z_n and z_f also define stereoscopicimaging limit regions.

As shown in FIG. 3, the display devices 42 and 52 are configured byarranging the parallax barriers 47 and 57 in front of the front surfacesof the flat display panels 46 and 56. A gap g is assured between theflat display panels 46 and 56 and the parallax barriers 47 and 57. Eachof the flat display panels 46 and 56 has a display screen having thevertical and horizontal directions. On this display screen, pixels eachhaving a predetermined width are arrayed in a matrix at a given pixelpitch pp. Each of the parallax barriers 47 and 57 is configured by alenticular sheet or slit sheet in a one-dimensional II system. On eachof the parallax barriers 47 and 57, a large number of cylindrical lensesor slits are arrayed at a lens pitch Pe along the horizontal direction(x direction). Or each of the parallax barriers 47 and 57 is configuredby a fly-eye lens sheet or pinhole sheet in a two-dimensional II system,and a large number of microlenses or pinholes are arrayed at lenspitches Pe(H) and Pe(V) along the horizontal direction (x direction) andvertical direction (y direction). These cylindrical lenses, slits,microlenses, or pinholes are called optical apertures or optical pupils.In a stereoscopic image display system based on the II system, the arraypitch Pe (horizontal pitch Pe(H) or vertical pitch Pe(V)) of the opticalapertures or optical pupils is set to be an integer multiple of thepixel pitch pp at which pixels are arrayed on the display screen.

Element image regions 60 are defined on the display screen of each ofthe flat display panels 46 and 56 by dividing and segmenting the displayscreen into regions facing the optical apertures or optical pupils. Thatis, in the stereoscopic image display system based on theone-dimensional II system, the regions 60 where element images aredisplayed are defined in correspondence with the respective cylindricallenses or slits, and the element image regions 60 are successivelyarrayed in the x direction. Also, in the stereoscopic image displaysystem based on the two-dimensional II system, the regions 60 whereelement images are displayed as a 2D image elements (a planar imageelements) are defined in correspondence with the respective microlensesor pinholes, and the element image regions 60 are successively arrayedin a matrix in the x and y directions. The element image regions 60 aredefined depending on an observation reference visual distance Lo and anobservation reference plane 62 on the observation reference visualdistance Lo, as references of a normal stereoscopic observation rangewhich are set for each of the display devices 42 and 52. Parallax imagesin various directions, which are captured by the multi-camera, aredistributed to the element image regions to display element images onthese element image regions 60. Please refer to a disclosure, forexample, in JP-A 2005-331844 (KOKAI), which describes details ofdistribution of the parallax images to the element image regions.

As for the multi-camera 1, as has already been described above, theimaging reference distance Lc and interval x_c are given by:

g:pp=Lc:x _(—) c

where pp is a pixel interval (pixel pitch) on each of the display panels46 and 56, and g is an interval (gap length) between each of theparallax barriers 47 and 57 and the display screen of each of thedisplay panels 46 and 56. When equation (1) holds, and the imagingreference distance Lc is set as the observation reference visualdistance Lo, a stereoscopic image having the same size as each of thereal objects 3-1 to 3-3 is formed in front of or on the back side ofeach of the display devices 42 and 52. The imaging reference distanceLc, at which a formation relationship of the stereoscopic image havingthe same size as each of the real objects 3-1 to 3-3 in front of or onthe back side of each of the display devices 42 and 52 is satisfied, iscalled the projection plane 2 shown in FIG. 2. Using this projectionplane 2 as an imaging reference plane, a high-resolution stereoscopicimage free from any distortion can be displayed based on captured imagedata on each of the display devices 42 and 52. The projection plane 2 asthe imaging reference plane is different from an imaging plane which isdefined by a distance to an object when an image of the object is infocus, and that object is formed as images on the imaging elements 30-1to 30-n. That is, the projection plane 2 is defined as the imagingreference plane which allows to acquire captured image data based onwhich a high-resolution stereoscopic image free from any distortion canbe displayed on each of the display devices 42 and 52.

The projection plane 2 corresponds to the imaging reference plane whichmatches the display screen of each of the display devices 42 and 52. Theimaging reference distance is set as the observation reference visualdistance of the three-dimensional display, and an imaging position isset as a viewpoint on the observation reference plane of thethree-dimensional display, thereby displaying an object to be capturedin a real scale. This projection plane 2 corresponds to the displayscreen displayed on each of the display devices 42 and 52 shown in FIG.1, and it is preferable to capture an image of an object with referenceto the projection plane 2 as the imaging reference plane. In place ofdisplaying the projection plane 2 in each of the display devices 42 and52, a numerical value associated with the imaging distance to theprojection plane may be set by another method, or another display, forexample, a display that requests to move the multi-camera forward orbackward from the current position may be made.

In the imaging optical system shown in FIG. 2, when ranges used asparallax images are edited by clipping within the imaging range 6 of themulti-camera 1, even when an image of any real object is captured, itcan be displayed in an enlarged scale as a stereoscopic image in thevicinity of the display screen of the three-dimensional display. Controlof the ranges used as parallax images will be described below withreference to FIGS. 4A and 4B.

FIG. 4A shows an image 8-1 captured by the imaging element 5-1 on theleft end of the multi-camera 1 shown in FIG. 2, and FIG. 4B shows animage 8-n captured by the imaging element 5-n on the right end of themulti-camera 1 shown in FIG. 2. The imaging elements 5-1 to 5-n eachhaving an imaging width W_s have the same size, and output parallaximages of the same size. An image of a real object (illustrated as “◯”),which is located at the projecting position separated by the distancez_n from the projection plane 2, appears at the right end on the imagingelement 5-1, as shown in FIG. 4A, and appears at the left end on theimaging element 5-n, as shown in FIG. 4B. Also, an image of a realobject (illustrated as “Δ”) on the projection plane 2, that is, on theimaging reference distance Lc, appears at the center on the imagingelements 5 without being shifted to the left or right in both FIGS. 4Aand 4B. Furthermore, an image of a real object (illustrated as “□”),which is located at the depth position separated by the distance z_f,appears at the left end on the imaging element 5-1 in FIG. 4A, andappears at the right end on the imaging element 5-n in FIG. 4B.

The parallax images captured by the imaging elements 5-1 to 5-n areprocessed by the image processing unit 40, and are clipped in accordancewith ranges to be displayed. A stereoscopic image can be displayed onthe display device 42 according to the clipped parallax images. FIGS. 4Aand 4B show ranges used as parallax images, that is, clipping regions7-1, 7-2, and 7-3 to be clipped to be used as parallax images, which arebounded by bold lines. Note that the clipping regions 7-1, 7-2, and 7-3are set on imaging planes of all the imaging elements 5-1 to 5-n to havethe same size. A central reference line 10 of the clipping regions 7-1,7-2, and 7-3 is shifted from a center 12 of each of the imaging elements5-1 and 5-n by a shift value s_n or s_f, which is defined by equation(2). When a real object (illustrated as “◯”, “Δ”, or “□”) as an objectto be clipped is specified, even when an image of any real object(illustrated as “◯”, “Δ”, or “□”) is captured, that captured image isdisplayed on the three-dimensional display to display a stereoscopicimage in the vicinity of the display screen. In this case, for the shiftvalue s_n (or s_f), equation (2) holds from a relationship of similartriangles:

s _(—) n:x _(—) c=z _(—) c:Lc

where x_c is an x-coordinate of each of the lenses 4-1 and 4-n withreference to the center (corresponding to the central reference line 10)of the camera 1.

In case of perspective projection, the real object (“◯”) at theprojecting position appears to have a large size, and the real object(“□”) at the depth position appears to have a small size. Therefore, abroad clipping range 7 is set for the real object (“◯”) at theprojecting position, and a narrow clipping range 7 is set for the realobject (“□”) at the depth position, thus allowing to display homeostaticsizes.

Images in the clipping ranges 7 are decomposed into pixel levels, andare assigned to the display screen of the display panel 46 as componentsof element images. On pixels of the three-dimensional display, which isconfigured by exit pupils and pixel groups on its back surface,information changes depending on observation angles, and these pixelsbehave as those which present parallax information, thus displaying astereoscopic image on the display panel 46. Therefore, the display panel46 displays a clip image of one of the objects 3-1 to 3-3, on which thephotographer focuses interest.

As described above, of the multi-viewpoint images 8-1 to 8-n to becaptured, regions used as parallax images, that is, the clipping regions7-1, 7-2, and 7-3 are selected. As long as images of the real objects3-1 to 3-3 are captured within the imaging range 6, a three-dimensionalimage can be displayed based on the captured images in the vicinity ofthe display screen. More specifically, the sizes of the clipping ranges7-1, 7-2, and 7-3 indicate those of images when they are displayed onthe three-dimensional display, and the shift value s_n or s_f of aposition for each viewpoint image of each of the clipping ranges 7-1,7-2, and 7-3 defines a depth or projecting distance to be displayed onthe display screen. In other words, the shift value s_n or s_f has acorrelation with a distance in the depth or projecting direction fromthe projection plane 2.

The method of controlling the clipping regions 7-1, 7-2, and 7-3 suffersa problem about an image distortion when the real object (illustrated as“◯” or “□”) other than the real object (illustrated as “Δ”) on theprojection plane 2 is displayed in the vicinity of the display screen.The layout of the multi-camera 1 shown in FIG. 2 should be designed toreflect the relationship between the pixels and exit pupils of thethree-dimensional display, as described above with reference to FIG. 3.However, when the real object in front of or on the back side of theprojection plane 2 is displayed in the vicinity of the display screen byadjusting its clipping range, since the observation reference visualdistance and imaging reference distance are different, a mismatch ofperspective degrees occurs in a strict sense. In display, the mismatchof perspective degrees appears as a distortion. When the distance z_n orz_f assumes a small value, such distortion is not conspicuous, but it ispreferable to display a display image without any distortion.Furthermore, when shift positions of the lenses 4-1 and 4-n with respectto the imaging elements 5 are fixed, imaging ranges overlap each otherat a certain imaging reference distance Lc of the projection plane 2.Hence, when an object to be captured is set in advance, it is desirableto capture an image of that object from a position at which theprojection plane 2 nearly matches that object to be captured, that is, aposition separated from the object to be captured by the imagingreference distance Lc. When the objects 3-1 to 3-3 to be mainlydisplayed are laid out to be separated from the projection plane 2 bytoo large distances, the following problems are posed in addition to theaforementioned problem of perspective degrees. As the first problem,upon execution of clipping in consideration of the shift value (s_n,s_f), the imaging range may become insufficient. In this case, theclipping range 7 with reference to “◯” in FIG. 2 corresponds to theimaging range. As the second problem, upon execution of clipping inconsideration of the shift value (s_n, s_f), a resolution may becomeinsufficient. In this case, since an image is clipped by the clippingrange 7 with reference to “□” in FIG. 2, a resolution may becomeinsufficient. From this viewpoint, it is preferable to adjust (correct)the imaging reference distance.

When the shift positions of the lenses 4-1 and 4-n with respect to theimaging elements 5 can be changed, the problem of the insufficientimaging range is not posed. However, the problem of a distortion due toa difference between the observation reference visual distance andimaging reference distance and that of the insufficient resolutioncannot be solved.

From the aforementioned viewpoints, as long as an object on which thephotographer or observer focuses interest is set on the projection planeas an object to be captured, even when that object to be captured isclipped, no mismatch of perspective degree occurs, and a display imagefree from any distortion can be displayed.

In the multi-camera system according to the embodiment shown in FIG. 1,either even when the photographer decides one of the objects 3-1 to 3-3to be mainly displayed or even when the observer of thethree-dimensional display decides an object to be mainly displayed, thephotographer can recognize and optimize the object. When thephotographer decides one of the objects 3-1 to 3-3 to be displayed, heor she specifies an object to be displayed via the input unit 45 withreference to contents displayed on the display device 42, so as todisplay a display object screen used to specify the object to bedisplayed, thereby settling the object to be displayed. As the latterexample in which the observer decides the object to be mainly displayed,a case is assumed wherein the three-dimensional display unit 52 isplaced at a remote place, and the observer inputs a designationinstruction of a range to be observed in more detail via the input unit55 while observing contents displayed on the three-dimensional displayunit 52. In response to the designation instruction of the range to beobserved in more detail, a display object screen used to specify theobject to be displayed may be displayed to settle the object to bedisplayed. On this display object screen, multi-camera imaging isexecuted to be free from any insufficient resolution after clipping bycorrecting the imaging distance of the multi-camera to the object to bedisplayed so as to be matched with the projection plane.

FIG. 6 is a flowchart required to carry out optimal imaging when thephotographer decides an object to be mainly displayed using the imageprocessing unit 40 shown as blocks in FIG. 5. In the image processingunit 40 shown in FIG. 5, a display unit 5 corresponds to the displaydevice 42 shown in FIG. 1, that is, the viewer provided to themulti-camera 1 like that of a digital camera. The multi-viewpoint imageprocessing sequence according to the embodiment will be described belowwith reference to FIGS. 5 and 6.

As described above, the layout of the multi-camera 1 reflects theconfiguration of the display unit in the stereoscopic image displayapparatus. Therefore, preferably, the multi-camera 1 is configured sothat the display screen of the viewer as the display unit 5 matches theprojection plane 2. Also, the multi-camera is designed to have thisconfiguration, thus improving usability of the multi-camera 1.

When the multi-camera 1 starts imaging (step S10), it is confirmedwhether or not an object to be mainly displayed is displayed as astereoscopic image in the vicinity of the projection plane 2 displayedwithin the viewer as the display unit 42 in an imaging start state(initial state) (step S11). If the object to be displayed is notdisplayed in the vicinity of the projection plane displayed within thisviewer, the imaging position is moved back or forth while observing theviewer as the three-dimensional image display device to search for animaging position where the object to be displayed is displayed on theprojection plane 2 (step S12). In a state in which the object to bedisplayed is roughly displayed on the projection plane 2, it is judgedwhether or not the object is displayed to have an appropriate displaysize (step S13). When the display size of the object to be displayed isto be adjusted in step S13, an instruction to change a display range(clipping range) is input to the multi-camera 1 (step S14). Morespecifically, processing for enlarging or reducing the display size ofthe object is executed while maintaining the shift values (s_n, s_f) ofthe clipping regions 7-1, 7-2, and 7-3.

In order to maintain perspective degrees upon enlarging or reducing theranges of the clipping regions 7-1, 7-2, and 7-3, data is fed back tothe imaging reference distance to adjust (correct) the imaging referencedistance and camera interval (step S15). FIGS. 7 and 8 show a state ofprocessing in which the imaging reference distance is changed from adistance L to a distance L′, and the imaging reference distance isadjusted based on this changed imaging distance.

FIGS. 7 and 8 show imaging positions of the imaging units 30-1 to 30-nrequired to display images free from any distortion when the clippingrange 7-1 is set to be a range narrower than the projection plane 2.When the width of the projection plane 2 is set to be Wt, and that ofthe clipping region 7-1, that is, the clipping range is set to be W_c,an imaging reference distance L′ and camera position x_c′ are changed tosatisfy equation (3) (step S15).

W _(—) c/W _(—) t=x _(—) c′/x _(—) c=L′/L

When the clipping range 7-1 is changed, as shown in FIG. 7, whilemaintaining the imaging position, an ideal imaging reference distance Lis set to be a short distance L′. Therefore, the projection plane 2after the clipping range 7-1 is changed is set on the front side, asshown in FIG. 8, compared to the previous projection plane 2 before theclipping range is changed, as shown in FIG. 7. As a result, az-coordinate of the object to be displayed, which is displayed on theviewer, is relatively moved to the back side. Thus, the photographer isinformed of an imaging position separated by a distance larger than theimaging reference distance, thereby prompting the photographer to changethe imaging position (that is, to move forward), so as to capture animage at the imaging reference distance.

With the aforementioned processing, the camera positions x_c of theimaging units 30-1 to 30-n are changed to camera positions x_c′. Thischange corresponds to that to imaging units 30-k to 30-m, which areselected from the imaging units 30-1 to 30-n and are used as validcaptured image data (k and m are integers which satisfy 1<k<m<n). Uponchanging to the imaging units 30-k to 30-m, images from the imagingunits 30-k to 30-m are interpolated to prepare parallax images as manyas the required number of parallax images in step S16. The interpolatedparallax images are preferably colored to images different from thenon-interpolated parallax images so as to clearly specify that they aregenerated by interpolation.

The photographer confirms that the imaging distance is changed, and theprojection plane 2 is changed within the display screen upon changing ofthe clipping range, as described above, and need only move themulti-camera position used in imaging to shorten a distance to theobject to be captured. Although it is ideal to change the camerapositions x_c which is expressed depending on camera coordinate, it isespecially difficult to narrow down the camera interval in terms of thestructure of the multi-camera. Therefore, it is preferable to leave theactual camera pitch of the multi-camera unchanged. Then, the camerapitch is left unchanged, and multi-viewpoint images imaging positions(x-coordinates) x_c′ are changed can be generated by image interpolationprocessing based on either an interpolation or extrapolation methoddepending on imaging conditions, and these multi-viewpoint images can beused (step S16). A screen is displayed using these interpolatedmulti-viewpoint images, and step S12 is executed according to thisdisplay screen.

Even when the object to be displayed can be displayed at a displayposition as a result of movement of the imaging position and imageinterpolation, as described above, it may not often fall within adisplay range in the depth direction of the three-dimensional displaydue to a large depth, that is, a large thickness of the object to bedisplayed (NO in step S17). In such case, when the imaging units 30-1 to30-n are shifted to reduce the camera coordinates x_c to, for example,½, as shown in FIG. 9, a space of the displayed state can be reducednearly to ½ in the z direction (step S18). In the multi-camera, since itis impossible to shift the imaging units 30-1 to 30-n in practice,captured image data at the ½ camera interval need only be prepared bythe image interpolation processing even for images captured at thecamera coordinates x_c.

If the object to be displayed falls within the display range in thedepth direction of the three-dimensional display in step S17, or if theobject to be displayed is adjusted to fall within the display range inthe depth direction of the three-dimensional display by the processingin step S18, image data from the imaging units 30-1 to 30-n or 30-k to30-m under this imaging condition are sorted to display images togenerate element image data, and are stored in a storage device (notshown) (step S19). The element image data are prepared in this way, thusending a series of processes (step S20). If necessary, the processesfrom step S20 are repeated again for detailed settings.

As shown in FIG. 5, the image processing unit which executes theaforementioned sequence includes a storage unit 20 which storesmulti-viewpoint images captured by the multi-camera 1, and an imagingcondition storage unit 24 which stores the imaging condition of themulti-camera 1. Following clipping conditions C1 to C3 for themulti-viewpoint images captured by the multi-camera 1 are stored in aclipping condition storage unit 22.

(C1) Clipping size (an initial value is 1 as a normalized value)

(C2) Interval of multi-camera 1 (an initial value is 1 as a normalizedvalue)

(C3) Imaging reference distance (which can be adjusted artificiallyusing the shift value (an initial value=0))

A parallax image generation unit 26 executes the image interpolationprocessing and clipping processing according to the position of themulti-camera 1 stored in this clipping condition storage unit 22, whenan image acquisition position is required to be change. Parallax imagesgenerated by this parallax image generation unit 26 are sorted forrespective pixels in a sort processing unit 28, and are changed to aformat to be displayed on the three-dimensional display. In this case,the format for the three-dimensional display means an element imagearray in which element images are arranged in a tile pattern.

The element image array generated by the sort processing unit 28 issupplied to the display unit 5, which then displays a three-dimensionalimage. The element image array supplied to the display unit 5 is alsosupplied to a display condition adjustment unit 32. While observing theimage displayed on the display unit 5, the clipping size and theposition (interval) of the multi-camera 1 are adjusted using the displaycondition adjustment unit 32. In this case, as has already beendescribed above, the adjustment of the position (interval) of themulti-camera 1 includes a virtual camera position or camera intervalwhich allows to acquire captured image data prepared by the imageinterpolation. After the adjustment of the display condition adjustmentunit 32, especially, after the clipping size is changed, the imagingcondition is reflected, and the imaging reference distance and camerainterval have to be corrected while maintaining similarity with thelayout of the imaging condition. The imaging reference distance can beartificially changed by adjusting the shift values of the clippingranges 7-1, 7-2, and 7-3. However, basically, it is prompted to changethe imaging reference distance in place of adjustment of the shiftvalues of the clipping ranges in the display control adjustment unit 32.Parameters adjusted by the display condition adjustment unit arereflected to contents in the clipping condition storage unit 24. As aresult, the display state of the three-dimensional image on the displayunit 5 is updated in real time.

In this case, when an image is displayed on the display unit 5 whilereflecting the display condition set by the display condition adjustmentunit 32, the imaging ranges of the multi-viewpoint images may becomeinsufficient. Images of regions having the insufficient imaging regionsmay be replenished by substitution processing using parallax informationof already acquired multi-viewpoint images. The observer may be informedof these substituted parallax images by coloring substituted parts toindicate substituted images.

The display condition adjustment unit 32 preferably include a trackingprocessing unit (not shown), which recognizes an object to be displayedin the vicinity of the display screen by image processing, and can trackthe object even when the imaging condition has changed and the object tobe captured has moved. Preferably, this tracking processing unit alwaysdisplays the object by automatically changing or updating the parametersstored in the clipping condition storage unit 22.

FIGS. 10 and 11 show the image processing unit 40, which allows theobserver of the three-dimensional display to decide the object to bemainly displayed, and the processing sequence in this image processingunit 40. The unit shown in FIG. 10 includes two displays 5-1 and 5-2, asshown in FIG. 1, unlike the processing unit shown in FIG. 5. One displayunit 5-1 corresponds to the viewer provided to the multi-camera 1, andthe other display unit 5-2 corresponds to the three-dimensional display.

As in the processing in the flow chat shown in FIG. 6, the multi-camera1 starts imaging (step S10). Initially, the observer observes athree-dimensional image in an imaging start state (initial state), andadjusts parameters. The observer adjusts the parameters while observingthe display unit 5-2. The adjustment parameters are displayed on thisdisplay unit 5-2, and the parameters are input to the display conditionadjustment unit 32 of the image processing unit 40 via the input unit 55and the transmission/reception units 48 and 58 (step S21). An image towhich parameters other than the shift amount as a parameter arereflected is displayed on the display unit 5-2. An image of the virtualprojection plane 2 converted from the shift amount is generated by CG.This projection plane 2 is colored and displayed in the display unit 5-2(step S22). Unlike the photographer, the observer cannot change theimaging reference distance L. Hence, even when the shift amount as theparameter is manipulated, the projection plane 2 is merely displayed onthe display screen of the three-dimensional display. The photographerobserves the same screen as that observed by the observer in the vieweras the display unit 42. When an object to be displayed is not displayedin the vicinity of the projection plane displayed in this viewer, theimaging position is moved back or forth while observing the viewer asthe three-dimensional image display device, thus searching for animaging position where the object to be displayed is displayed on theprojection plane 2 (step S12).

The observer judges whether or not the object is displayed to have anappropriate display size in a state in which the object to be displayedis roughly displayed on the projection plane 2 as a result of movementof the photographer (step S13). When the observer wants to adjust thedisplay size of the object to be displayed in step S13, he or she inputsan instruction to change a display range (clipping range) to the imageprocessing unit 40 via the input unit (step S14). More specifically,processing for enlarging or reducing the display size of the objectwhile maintaining the shift values (s_n, s_f) of the clipping regions7-1, 7-2, and 7-3 is executed.

When the ranges of the clipping regions 7-1, 7-2, and 7-3 are to beenlarged or reduced, in practice, data is fed back to the imagingreference distance to adjust the imaging reference distance and camerainterval (step S15), as has been described above with reference to FIGS.7 and 8. After that, image interpolation processing upon change of thereference imaging distance is executed, and multi-viewpoint images whoseimaging positions (x-coordinates) x_c′ are changed are generated (stepS16). A screen is displayed using the interpolated multi-viewpointimages, and step S12 is executed according to this display screen.

If the object does not fall within the display range in the depthdirection of the three-dimensional display due to a large depth, thatis, a large thickness of the object to be displayed (NO in step S17),the camera coordinates x_c are shifted, as shown in FIG. 9, thusreducing (compressing) a space of the displayed state in the z direction(step S18). In the multi-camera, since it is impossible to shift theimaging units 30-1 to 30-n in practice, captured image data at the ½camera interval need only be prepared by the image interpolationprocessing even for images captured at the camera interval x_c.

If the object to be displayed falls within the display range in thedepth direction of the three-dimensional display in step S17, or if theobject to be displayed is adjusted to fall within the display range inthe depth direction of the three-dimensional display by the processingin step S18, image data from the imaging units 30-1 to 30-n or 30-k to30-m under this imaging condition are sorted to display images, so as togenerate element image data, and are stored in a storage device (notshown) (step S19). The element image data are prepared in this way, thusending a series of processes (step S17). If necessary, the processesfrom step S21 are repeated again for detailed settings.

Note that the input clipping sizes of the clipping regions and thecamera interval, which changes in synchronism with the sizes, and thecamera interval manipulation in the compression display processing ofthe depth method are taken into consideration, and they are stored inthe clipping condition storage unit 22 as clipping conditions. In thisprocessing, the shift values are not reflected. This is because a shiftinstruction is required to be displayed on the display unit to promptthe photographer to move the imaging position. At this time, as forwhich part of the object to be captured is to be displayed, a shiftvalue to be displayed on the display screen, that is, the imagingreference distance is detected based on the shift value manipulated bythe observer using the display condition adjustment unit 32. Forexample, as shown in FIG. 12, a virtual plane 9 of the object to becaptured is display by means of CG. The photographer can move theimaging position back or forth so that this plane 9 of the object to becaptured matches the projection plane.

With the aforementioned method, a divergence of the imaging referencedistance from an ideal value, which causes a distortion inthree-dimensional image display is signaled to the photographer to givea guide to move to an ideal imaging reference distance.

Note that details of the image interpolation processing have not beendescribed. For example, an existing method such as a known bilinearmethod or bicubic method need only be used. Also, the camera may beexpected to have functions such as zoom-in/out, lens shift, and movementof a focal length, and it is apparent that the present embodiment can beapplicable to these operations.

As described above, there can be provided the captured image acquisitionmethod, which allows to acquire multi-viewpoint images required todisplay an object to be displayed free from any distortion within adisplay range of a three-dimensional display having a parallax barrier,and is required to display a stereoscopic image, and a method andapparatus for displaying a stereoscopic image from the acquired images,can be provided.

According to a stereoscopic image display apparatus which captures astereoscopic image of the present embodiment, in the method ofacquiring, by actual imaging, and displaying multi-viewpoint images fora three-dimensional display based on the parallax barrier system, aphotographer is informed of an appropriate imaging reference distance toappropriately display a desired object to be displayed within a displayrange of the three-dimensional display.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

1. A stereoscopic image display apparatus comprising: a firstthree-dimensional display device includes, a first display unitconfigured to display a 2D image, the 2D image including elementalimages, and a first light control unit configured to control directionsof light rays emitted from the first display unit, wherein the firstdisplay unit displays each of the elemental images in a first specificarea determined with the directions of the controlled light rays, so asto display a three dimensional image; a multi camera configured tocapture multi-viewpoint images of a real object from specific viewpoints placed at certain intervals, wherein the multi-camera has aprojection plane serving as an imaging reference plane; an imageprocessing unit configured to process images taken by the multi camera,the image processing unit including; a parallax image generation unitconfigured to generate parallax image data including clip parallaximages having specified ranges which are clipped from themulti-viewpoint images based on information about clipping ranges of themulti-viewpoint images, a sort processing unit configured to sort pixelsfrom the clip parallax images and rearrange the sorted pixels togenerate the elemental images, and a display condition adjustment unitconfigured to adjust parameters required to display thethree-dimensional image, wherein the parameters are so adjusted as tocapture the real object as the multi-viewpoint images having a desiredsize at the vicinity of the projection plane with reference to thedisplayed three-dimensional image, wherein the image processing unit isconfigured to correct the imaging reference distance and the intervalsof the view points based on the adjusted parameters and the imagingcondition to derive a corrected imaging reference distance and acorrected interval, which are required to display the three-dimensionalimage without any distortion.
 2. The stereoscopic image displayapparatus according to claim 1, further comprising: a secondthree-dimensional display device includes; a second display unitconfigured to display the 2D image, and a second light control unitconfigured to control directions of light rays emitted from the seconddisplay unit, wherein the second display unit displays each of theelemental images in a second specific area determined with thedirections of the controlled light rays, so as to display the threedimensional image; and an input unit configured to input an instruction.3. The stereoscopic image display apparatus according to claim 2,wherein the display condition adjustment unit adjusts the parametersbased on a clipping condition, when the instruction including theclipping condition is input to the input unit, the parallax imagegeneration unit generates the parallax image data including the clipparallax images in accordance with the adjusted parameters, and theimage processing unit calculates a virtual projection plane based on theinput clipping condition and causes the second display unit to displaythe virtual projection plane.
 4. The stereoscopic image displayapparatus according to claim 3, wherein the parallax image generationunit generates the parallax image data which includes the parallaximages reflecting the display condition, wherein the parallax imagesinclude interpolation images each of which is calculated fromsubstituting the parallax images adjacent to each other, when themulti-viewpoint images is insufficient for displaying thethree-dimensional image.
 5. The stereoscopic image display apparatusaccording to claim 4, wherein the display condition adjustment unitincludes a tracking processing unit which is configured to recognize thereal object to be captured to track the real object, and the trackingprocessing unit automatically updates the parameters for the clippingcondition and causes the first three-dimensional display device todisplays the three dimensional image, even when the imaging conditionhas been changed, and the real object to be captured has moved.